Skip to content

Inflection-208 Add multisource and DMLex dictionary parsing support - #212

Open
nciric wants to merge 7 commits into
unicode-org:mainfrom
nciric:Inflection-208-dmlex-multisource-parser
Open

nciric wants to merge 7 commits into
unicode-org:mainfrom
nciric:Inflection-208-dmlex-multisource-parser

Conversation

@nciric

@nciric nciric commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Fixes #208

Summary

Adds support for parsing OASIS DMLex v1.0 JSON dictionaries alongside (or instead of) Wikidata lexeme dumps in tools/dictionary-parser, with configurable multi-source conflict resolution.

Key Changes

  • Explicit Multi-Source CLI Flags (ParserOptions):
    • --wikidata <file1> [<file2> ...] and --dmlex <file1> [<file2> ...] (supporting multiple space- or comma-separated files as well as --wikidata=<files> / --dmlex=<files>, including .bz2 compressed inputs).
    • --merge-conflicts[={true|false}] (default false): controls whether same-part-of-speech entries across multiple input files are overridden by later files (false) or merged (true).
    • Replaced static in-place mutation of Grammar.TYPEMAP with per-instance overrides (customTypeMap and ignoredGrammemes) on ParserOptions.
  • OASIS DMLex v1.0 Deserialization & Streaming (DMLexEntry, DMLexInflectedForm, DMLexStringListDeserializer, ParseWikidata):
    • Streaming Jackson parser (parseDMLexStream) validating the top-level DMLex JSON object (langCode and entries) and converting entries into the shared Lemma / Inflection pipeline.
    • Supports both flat string tags and nested DMLex tag/label/pronunciation objects, synthesizing a default form from headword when inflectedForms is omitted.
    • Extended Grammar lookup (getMappedGrammemes) to normalize camelCase, snake_case, and hyphenated DMLex/Universal Dependencies grammatical tags in addition to Wikidata Q-IDs.
  • Two-Tier Cross-Source Conflict Resolution (ParseWikidata, DictionaryEntry):
    • Tier 1 (Lemma/Paradigm level): When --merge-conflicts=false (default) and multiple source files are provided, stages lemmas and preempts any earlier (lemma, PartOfSpeech) paradigm superseded by a later source file before generating inflection patterns (inflectional.xml) or surface forms (dictionary.lst), preventing orphaned surface forms and inflated pattern counts.
    • Tier 2 (Surface-form level): Tracks per-PartOfSpeech provenance (sourceIndex) inside DictionaryEntry so same-POS surface-form collisions from a later source replace earlier ones while preserving non-conflicting parts of speech (e.g., keeping interjection from an earlier source when a later source adds noun) and intra-file homographs.
  • Testing (ParseWikidataTest):
    • Added unit tests and test fixtures covering standalone DMLex parsing, multi-source Wikidata + DMLex override (--merge-conflicts=false), cross-source union merging (--merge-conflicts=true), partial multi-POS overrides, space/comma-separated file lists, CLI option compatibility (--map-grammeme, --expand-grammemes, --ignore-property, --ignore-entries-with-grammemes, --exclude-language), and schema validation errors.

@nciric
nciric requested a review from grhoten October 9, 2026 23:00

@grhoten grhoten left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's going in the right direction. I'm getting hung up on some of the details. Some of the details I've gotten hung up on for years, especially the layering/overriding of data.

Comment on lines +134 to +150
/**
* Returns {@code true} when multiple input files are provided and {@code --merge-conflicts} is {@code false}
* (the default). In this mode, later files on the CLI override conflicting entries of the same
* {@link PartOfSpeech} from earlier files via a two-tier strategy:
* <ul>
* <li><b>Tier 1 (Lemma/Paradigm level)</b>: {@link #recordOrAnalyzeLemma} stages lemmas and
* {@link #processStagedLemmas} strips any {@code (lemma.value, PartOfSpeech)} superseded by a later
* source before {@link #analyzeInflections} runs. This prevents removed inflected surface forms
* (e.g., an obsolete plural dropped in a patch file) from lingering as orphans in {@code dictionary.lst}
* and prevents superseded paradigms from inflating {@link InflectionPattern} counts in {@code inflectional.xml}.</li>
* <li><b>Tier 2 (Surface-form level)</b>: {@link DictionaryEntry#merge} resolves remaining surface-form
* collisions per {@link PartOfSpeech} using {@code sourceIndex}.</li>
* </ul>
*/
private boolean isMultiSourceOverwriteMode() {
return !parserOptions.mergeConflicts && parserOptions.sourceInputs.size() > 1;
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not quite understanding this logic. What does merge mean? Here are the options that I see when there is a conflict.

  1. Pattern is mixed in with the words with normal priority sorting.
  2. Pattern is mixed in, and the conflict comes first.
  3. New pattern and grammemes overrides all previous data for a surface form.
  4. Normal processing happens, until you hit the overrides, and then one of the above options happens.

@nciric nciric Oct 15, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If merging continues being an issue, we can remove it from this round and just do overrides for now.

Option 4 makes the design much cleaner. I removed the two-tier lemma staging (stagedLemmas) in ParseWikidata and the per-POS PosSlice splitting in DictionaryEntry, and replaced them with Option 4:

  • All files now stream directly into analyzeLemma(...) with normal intra-file merging.
  • When a surface form from a later source file (entry.sourceIndex > this.sourceIndex) collides with an existing DictionaryEntry:
    • By default (--merge-conflicts=false — Option 4 + Option 3): The new entry's pattern and grammemes completely override the previous data for that surface form (matching how --supplementalfile works in buildDictionary). If the same override file itself has multiple entries for that surface form (same sourceIndex), they merge normally with each other. Before writing inflectional.xml, DocumentState.printDocument() prunes any InflectionPattern whose surface forms were all overridden so inflectional.xml stays in sync with dictionary.lst.
    • When --merge-conflicts=true (Option 4 + Option 2): The new entry's grammemes and pattern are mixed in, and the later source's conflicting pattern is placed first in priority ahead of earlier sources' patterns.

Comment on lines +1181 to +1288
return TYPEMAP.get(grammeme);
if (grammeme == null) {
return null;
}
Set<? extends Enum<?>> mapped = TYPEMAP.get(grammeme);
if (mapped != null) {
return mapped;
}
return TYPEMAP.get(normalizeGrammemeKey(grammeme));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is hit constantly, and it can add to the parse time. Is there a way to optimize this. Perhaps there should be 2 maps? If that complicates the logic elsewhere too much, then maybe not.

Also if grammeme is null, that't an error condition. You need something! Even a part of speech helps.

@nciric nciric Oct 15, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Removed normalizeGrammemeKey() completely; getMappedGrammemes() is back to a single TYPEMAP.get(grammeme) lookup with zero string allocations, and DMLex entries use canonical grammeme names from DEFAULTMAP (seeded once into TYPEMAP at static initialization).
  • Added an explicit if (grammeme == null) throw new IllegalArgumentException("Grammeme cannot be null"); check so a null grammeme fails fast.

Comment on lines +68 to +75
// Per-run string-to-grammeme overrides configured via --map-grammeme or Q-ID arguments to
// --ignore-property / --ignore-entries-with-grammemes. Stored per ParserOptions instance so
// static Grammar.TYPEMAP is never mutated in place.
private final Map<String, Set<? extends Enum<?>>> customTypeMap = new HashMap<>();
// Per-run enum overrides configured via --ignore-property or --ignore-entries-with-grammemes
// when specified by grammeme name (e.g. "countable", "shortForm"). Any resolved grammeme enum
// present in this map is replaced with its configured Grammar.Ignorable token during lookup.
private final Map<Enum<?>, Grammar.Ignorable> ignoredGrammemes = new HashMap<>();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you elaborate on the purpose of this change? What issue was encountered when mutating Grammar.TYPEMAP?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was originally added out of concern that mutating static Grammar.TYPEMAP via --map-grammeme or --ignore-property across multiple ParseWikidata.main(...) calls in ParseWikidataTest could affect subsequent tests in the same JVM.

In practice, the test cases use distinct test-only grammemes and keeping separate customTypeMap and ignoredGrammemes fields on ParserOptions made both ParserOptions and every grammeme lookup unnecessarily complicated. I removed customTypeMap and ignoredGrammemes and reverted ParserOptions back to mutating Grammar.TYPEMAP directly as before.

Comment on lines +381 to +390
var grammemeEnum = DEFAULTMAP.get(grammeme);
if (grammemeEnum == null) {
grammemeEnum = DEFAULTMAP.get(Grammar.normalizeGrammemeKey(grammeme));
}
if (grammemeEnum == null && !grammeme.matches("Q\\d*")) {
Set<? extends Enum<?>> mapped = Grammar.getMappedGrammemes(grammeme);
if (mapped != null && mapped.size() == 1) {
grammemeEnum = mapped.iterator().next();
}
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is getting complicated. Is there a way to simplify this?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Simplified—by removing customTypeMap, ignoredGrammemes, ParserOptions.getMappedGrammemes(), and normalizeGrammemeKey(), setIgnoreProperty() and toEnumSet() are back to their original logic from main.

Comment on lines -67 to -73
void reset() {
value = null;
isRare = false;
grammemes.clear();
inflections.clear();
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was done to improve the performance. It's not stored long term. Can you elaborate on why you made this change?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lemma.reset() was only removed because the earlier two-tier override logic buffered Lemma instances in stagedLemmas until all files were read. Now that we use streaming (Option 4) without stagedLemmas, Lemma is short-lived again and I have restored Lemma.reset().

* unusual case where {@code "entries"} appears before {@code "langCode"} in the JSON key order,
* entries are buffered in {@code entriesBeforeLangCode} until {@code "langCode"} is read.
*/
private void parseDMLexStream(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a way to leverage a JsonDeserializer to simplify this logic? This seems to be going fairly low level with the parsing.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done—replaced the manual JsonToken loop in parseDMLexStream with objectMapper.readValue(parser, DMLexEntry.Lexicon.class) and simplified DMLexStringListDeserializer to use JsonNode.

@nciric

nciric commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor Author

@grhoten I did a regression check, seems no differences between old and new parser (and all tests pass).

I've checked all 40 languages in resources/org/unicode/inflection/dictionary/ on wikidata-20260902-lexemes.json.bz2 (comparing the output of main against this branch using each language's generated with options: flags) and found:

  • 39 out of 40 languages have 0 differences in dictionary_<lang>.lst and 0 differences in <pattern> and <inflection> element ordering in inflectional_<lang>.xml.
  • Russian (ru) differs by 1 word out of ~916,600 entries in dictionary_ru.lst (дававши: imperfective long-form past gerund participle verb inflection=1eb -> дававши: imperfective past gerund participle verb inflection=1eb, along with its two <stem/>вавши lines in pattern 1eb of inflectional_ru.xml).
    • Why ru differs by 1: dictionary_ru.lst is generated with --ignore-property countable,indefinite,long-form,attributive,pejorative,uncountable,long-form. On main, ParserOptions.setIgnoreProperty() only checked grammemeEnum.name().equalsIgnoreCase(grammeme), where FormType.LONG_FORM.name() is "LONG_FORM" (with an underscore), so --ignore-property long-form silently failed to match. Because setIgnoreProperty() now also checks grammemeEnum.toString().equalsIgnoreCase(grammeme), --ignore-property long-form takes effect as intended.

@grhoten

grhoten commented Oct 22, 2026

Copy link
Copy Markdown
Member

Why ru differs by 1: dictionary_ru.lst is generated with --ignore-property countable,indefinite,long-form,attributive,pejorative,uncountable,long-form. On main, ParserOptions.setIgnoreProperty() only checked grammemeEnum.name().equalsIgnoreCase(grammeme), where FormType.LONG_FORM.name() is "LONG_FORM" (with an underscore), so --ignore-property long-form silently failed to match. Because setIgnoreProperty() now also checks grammemeEnum.toString().equalsIgnoreCase(grammeme), --ignore-property long-form takes effect as intended.

It looks like this different will have to be addressed.

@nciric

nciric commented Oct 23, 2026

Copy link
Copy Markdown
Contributor Author

Why ru differs by 1: dictionary_ru.lst is generated with --ignore-property countable,indefinite,long-form,attributive,pejorative,uncountable,long-form. On main, ParserOptions.setIgnoreProperty() only checked grammemeEnum.name().equalsIgnoreCase(grammeme), where FormType.LONG_FORM.name() is "LONG_FORM" (with an underscore), so --ignore-property long-form silently failed to match. Because setIgnoreProperty() now also checks grammemeEnum.toString().equalsIgnoreCase(grammeme), --ignore-property long-form takes effect as intended.

It looks like this different will have to be addressed.

It seems there was a bug in the old code where long-form parameter was not obeyed, and ru-RU was exposing it. I can easily revert the fix and make 0 differences. WDYT?

Comment on lines +288 to +289
if (grammemeEnum.name().equalsIgnoreCase(grammeme)
|| grammemeEnum.toString().equalsIgnoreCase(grammeme)) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I think this should change to grammemeEnum.toString, and not use grammemeEnum.name.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@grhoten

grhoten commented Nov 12, 2026

Copy link
Copy Markdown
Member

I had AI review the conformance of the spec on these changes. Please take these comments into consideration. If the part of speech is "n-masc", it would be a noun part of speech with a grammatical gender of masculine.

DMLex v1.0 spec conformance

The parser currently accepts a DMLex-like JSON format rather than the DMLex v1.0 JSON serialization (spec §5.2, dmlex.schema.json). Several core members are named or interpreted differently from the spec, and spec-conformant files are mostly dropped, some of them silently.

DMLex v1.0 This PR Effect on a spec-conformant file
entry.partsOfSpeech: array of tag strings DMLexEntry binds only partOfSpeech / pos partsOfSpeech is silently ignored because of @JsonIgnoreProperties(ignoreUnknown = true), so entries lose their part of speech
inflectedForm.tag: one code (e.g. sg-gen) whose meaning is defined by inflectedFormTags tag is merged into labels, and each label must be a single Inflection grammeme name sg-gen → "is not a known grammeme" → the whole entry is dropped, with only a line on stderr
Headword grammatical properties (gender, aspect) are folded into the POS tag, e.g. n-masc, v-perf (partOfSpeech object type) Gender goes in entry labels Spec-style gender is lost along with the POS
labels are usage restrictions (temporal, regional, register, domain, e.g. colloq, neo) labels carry case, number and gender Real usage labels fail the grammeme lookup and drop the entry
partOfSpeechTags, inflectedFormTags, labelTags explain the tags and map them to external inventories via sameAs (§A.1.6, §A.1.7) DMLexEntry.Lexicon reads only langCode and entries The spec's mapping mechanism is ignored
pronunciations[].transcriptions[] with text and scheme (an IETF tag such as en-fonipa) scheme is ignored and every transcription is passed to addSound("P898", …) as IPA Non-IPA transcriptions are treated as IPA

As a result:

  • The spec's own example A.28 (Irish folúsghlantóir, with partsOfSpeech: ["n-masc"] and forms tagged sg-gen / pl) yields no dictionary entries.
  • The test fixtures don't validate against dmlex.schema.json, which uses additionalProperties: false:
    • Entry: partOfSpeech (and "partOfSpeech": "noun" as a bare string); per-entry langCode (only lexicographicResource has one).
    • Inflected form: value and id (the only allowed members are tag, text, labels, pronunciations); object-valued labels such as { "tag": "singular" } (labels are strings).
    • Pronunciation: a flat { "transcription": …, "scheme": "IPA" } (the spec uses transcriptions[], and scheme must be an IETF language tag).
  • DMLex commonly lists only the non-headword forms (A.28 lists only sg-gen and pl). The PR synthesizes a headword form only when inflectedForms is empty, so in that case the headword would be missing from the paradigm. Either require the headword as an explicitly tagged form, or define how it gets synthesized.

Suggested changes

  1. Model the spec's JSON objects directly. Use plain typed classes: List<String> for partsOfSpeech and labels, and Pronunciation / Transcription classes with text and scheme. Drop the alternate names (pos, forms, value, inflection, features, grammaticalFeatures, tags) and DMLexStringListDeserializer.

  2. Resolve tags through the Controlled Values module. Map each partsOfSpeech value, form tag and label through its *Tag definition's sameAs URIs. Wikidata entity URIs can be looked up directly in the existing Q-ID TYPEMAP, so multi-feature tags need no extra code. That also removes the need to seed DEFAULTMAP into TYPEMAP in Grammar, and DMLex files no longer depend on Inflection's internal grammeme spellings.

    {
      "langCode": "en",
      "entries": [{
        "id": "book-n",
        "headword": "book",
        "partsOfSpeech": ["n"],
        "inflectedForms": [
          { "tag": "sg",     "text": "book" },
          { "tag": "sg-gen", "text": "book's" },
          { "tag": "pl",     "text": "books" },
          { "tag": "pl-gen", "text": "books'" }
        ]
      }],
      "partOfSpeechTags": [
        { "tag": "n", "description": "noun",
          "sameAs": ["http://www.wikidata.org/entity/Q1084"] }
      ],
      "inflectedFormTags": [
        { "tag": "sg", "description": "singular", "for": "n",
          "sameAs": ["http://www.wikidata.org/entity/Q110786"] },
        { "tag": "sg-gen", "description": "singular genitive", "for": "n",
          "sameAs": ["http://www.wikidata.org/entity/Q110786", "http://www.wikidata.org/entity/Q146233"] },
        { "tag": "pl", "description": "plural", "for": "n",
          "sameAs": ["http://www.wikidata.org/entity/Q146786"] },
        { "tag": "pl-gen", "description": "plural genitive", "for": "n",
          "sameAs": ["http://www.wikidata.org/entity/Q146786", "http://www.wikidata.org/entity/Q146233"] }
      ]
    }

    A gendered noun tag works the same way, e.g. n-masc → sameAs Q1084 (noun) + Q499327 (masculine).

  3. Require exactly one part of speech per entry. This tool builds one paradigm per lemma and part of speech, so an entry with no part of speech, or with more than one, should be an error. DMLex makes partsOfSpeech optional and allows several for exceptional cases (e.g. nationality words like "Czech" used as both noun and adjective). Currently analyzeDMLexEntry accepts an entry with no part of speech, and adds every resolved part of speech to lemma.grammemes for an entry with several.

    Check after sameAs resolution, not by counting tags, because one tag can resolve to several grammemes. For example, n-masc resolves to noun (part of speech) and masculine (grammatical gender). The gender stays a lemma-level grammeme and finalizeAndAddInflection applies it to each inflected form, as it already does for Wikidata's P5185. The rule is countGrammemeType(lemma.grammemes, PartOfSpeech.class) == 1, with the file name and entry id in the error. Lexicographers can split multi-POS words into separate entries, one per part of speech.

  4. Fail loudly on bad input. DMLex files are hand-curated, so errors should stop the run instead of being logged and skipped:

    • List the spec members the parser doesn't use (senses, homographNumber, etymologies, placeholderMarkers, …) as explicitly ignored.
    • Fail on any other unknown property, on any tag with no *Tag definition, and on any sameAs that doesn't resolve to a grammeme.
    • Report the file name and entry id, and exit non-zero.
  5. Respect the transcription scheme. Only treat a transcription as IPA when its scheme has the fonipa variant subtag, or when it has no scheme (the spec allows the scheme to be implicit).

  6. Tests. Add the spec's A.28 example, with sameAs URIs added, as a fixture, and rewrite the existing fixtures so they validate against dmlex.schema.json.

@nciric

nciric commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review! I updated the DMLex parser (and extracted it into DMLexParser) to strictly follow the DMLex v1.0 JSON schema (dmlex.schema.json):

  • Modeled the spec objects directly and removed the custom aliases and deserializer.
  • Resolved partsOfSpeech, inflectedForms[].tag, and labels via sameAs Wikidata URIs in the Controlled Values tags (and removed DEFAULTMAP seeding in Grammar).
  • Required exactly one part of speech per entry, and required the headword to be present as a tagged form whenever inflectedForms is non-empty.
  • Enabled strict unknown-property checking and made invalid/unmapped tags or sameAs URIs fail immediately with the file name and entry id.
  • Restricted IPA matching to transcriptions with a fonipa subtag or an omitted scheme.
  • Added the A.28 example fixture and updated all test fixtures to validate against dmlex.schema.json.

Comment on lines +28 to +31
private final List<InflectionPattern> inflectionalPatterns;
private final List<InflectionPattern> rareInflectionalPatterns = new ArrayList<>();
private int sourceIndex;
private Map<InflectionPattern, Integer> patternSourceIndices = null;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's some redundancy here, and it can make it hard to scale this tool up.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted DictionaryEntry completely back to main (removing sourceIndex, patternSourceIndices, and the custom conflict-merge logic), and also dropped the redundant sourceFilenames list in ParserOptions.

Comment on lines +66 to +82
System.err.println("Usage: ParseLexicon [OPTIONS] <file1.xml or file1.lexicon>[ <file2.xml or file2.lexicon> ...]");
System.err.println("Usage: ParseLexicon [OPTIONS] (--wikidata <file1> [<file2> ...] | --dmlex <file1> [<file2> ...]) ...");
System.err.println("\nOPTIONS");
System.err.println(WIKIDATA_FILE + " <file1> [<file2> ...]\tOne or more Wikidata lexeme JSON/bz2 files (space- or comma-separated)");
System.err.println(DMLEX_FILE + " <file1> [<file2> ...]\tOne or more OASIS DMLex JSON files (space- or comma-separated)");
System.err.println(MERGE_CONFLICTS + "[={true|false}]\tWhen true, merge conflicting cross-file surface forms (with the later source's pattern first) instead of overwriting, default: false");

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The --wikidata and --dmlex options are great additions. I'm struggling with the --merge-conflicts option review. It's additional functionality that may be better to separate out to a different pull request because these changes are so big.

The supplemental data within this tool is fully merging the grammemes. The supplemental data in resources in the main build is replacing, but it's hard to reference patterns because the identifiers are not stable. The patterns are prioritized when the patterns differ by case because the non-lowercased variants are lowercased as an additional entry, but they should be the non-default. There are several ways to merge, and it needs thought on the right way to do it, especially with how you intend to actually use it.

I'd rather have the --wikidata and --dmlex argument order to define the inflection pattern order without a merge conflict option when all things are considered equal. This code originally would prioritize separate lexicon files with the first definition when there is ambiguity, when all other things are considered equal.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed --merge-conflicts and restored the original DictionaryEntry / DocumentState merge behavior from main:

  • --wikidata and --dmlex inputs are processed in CLI argument order, merging grammemes and inflection patterns across sources.
  • When rarity and pattern word counts (InflectionPattern::getCount) are equal, stable sorting on DictionaryEntry.inflectionalPatterns preserves CLI encounter order so the first definition's pattern comes first (while higher-frequency and non-rare patterns still take precedence otherwise).
  • Added unit tests covering --wikidata / --dmlex argument order (equal counts, unequal counts, and rare vs. non-rare) and verified zero regressions on wikidata-20261111-lexemes.json.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add support for lexical dictionary format for custom vocabulary

2 participants